PATENT ABSTRACTS OF JAPAN 

(1 1 publication number : 1 1-339060 

(43)Date of publication of application : 10.12.1999 



(51)Int.CI. G06T 13/00 

G06T 11/80 

G10H 1/00 

G10H 1/40 



(21 Application number : 10-272808 (71 Applicant : YAMAHA CORP 

(22)Date of filing : 28.09.1998 (72)Inventor : MIYAKI TSUTOMU 

SUZUKI HIDEO 
ISOZAKI YOSHIMASA 
SEKINE SATOSHI 



(30)Priority 

Priority number : 10 93867 Priority date : 24.03.1998 Priority country : JP 



(54) MUSICAL SOUND AND IMAGE GENERATING DEVICE 

(57)Abstract: 

PROBLEM TO BE SOLVED: To attain the 3D animation display of performance of 
each musical instrument or part synchronously with performance information. 
SOLUTION: This device is provided with a data base constituted of operation 
parts including operation information for storing the locus of the performing 
operation of a performance pattern subdivided for each musical instrument or 
each partand a sounding point marker for specifying the timing of sounding in the 
operation information. Thenthe operation part corresponding to the performance 
information is successively read from the data baseand basic operation information 
is generatedand a 3D animation image synchronizing with the performance 
information is generated based on the basic operation informationand displayed on 
an image displaying part 7. The display image can be arbitrarily selected by musical 
instrument change switches 44player change switches 45and stage change 
switches 47. Alsothe image from the arbitrary point of view can be displayed by a 
view point change switches 48. 



CLAIMS 



[Claim(s)] 

[Claim 1]Musical tone and an image generating device characterized by comprising 
the following. 

A musical sound generation part which generates musical tone based on 
performance information. 

An image generation part which generates image data which shows a situation of a 
performance corresponding to said performance information in a selected musical 
instrument or a part synchronizing with said performance information. 

[Claim 2]Have the part database built with operation parts including performance 
information which memorized a locus of a playing action of a performance pattern 
in which it was subdivided for every musical instrument or every partand said 
image generation partsaid musical tone according to claim 1 generating 3D 
animation image data corresponding to the performance information concerned 
based on information acquired by reading operation parts corresponding to said 
performance information from this part databaseand connecting these operation 
parts that carried out reading appearance one by oneand an image generating 
device. 

[Claim 3]Said musical tone according to claim 2 and an image generating 
devicewherein said operation parts have a pronunciation point marker in which 
timing of performance information showing a locus of a playing action of said 
subdivided performance pattern and pronunciation is shown. 
[Claim 4]Said musical tone according to claim 3 and an image generating 
devicewherein change of a performance character and a viewpoint in said 3D 
animation image data generated is enabled by operator. 

[Claim 5]Said musical tone according to claim 4 and an image generating device 
having a means to amend said performance informationaccording to change of said 
performance character and a viewpoint. 

[Claim 6]Even if said image generation part is a time of tempo of musical tone 
generated based on said performance information being changedSaid musical tone 
according to claim 1 and an image generating device currently making as 
[ generate / said image data set constant / image restoration speed of a portion 
corresponding to pronunciation operation in the image data concerned ]. 
[Claim 7]Said musical tone according to claim 1 and an image generating 
devicewherein said image generation part is made as [ embellish / the image data 
to generate ] based on said performance information or a pronunciation state of a 
musical sound generation part. 

[Claim 8]Said musical tone according to claim 1 and an image generating 
devicewherein said image generation part is made as [ set / the number of 
reproduction frames ] for every said selected musical instrument or part. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention]This invention relates to the musical tone and the image 
generating device which can display the playing configuration with 3D animation 
picture while generating musical tone. 
[0002] 

[Description of the Prior Art]Conventionallyin electrohone etc.performing an 
automatic performance according to desired automatic performance patternssuch 
as an automatic rhythm performance or an automatic base code performanceis 
known. That isbased on the chord specified when a player specified a chord (code) 
one by one according to advance of a musical piece about a code backing part or 
a base parta code backing sound and a base sound are automatically pronounced 
according to a predetermined performance pattern. On the other handas for the 
performance of the percussion instrument partthe normal pattern and the variation 
pattern are prepared for every various rhythms. 

One of patterns (style) can be chosen and an automatic performance can be 
carried out now. 

Two or more said variation patterns may be prepared not only in one piece. 
Generallysuch a performance pattern has the length for 1 - a number vibrant 
tuneand is made as [ perform / a continuous automatic rhythm performance ] by 
repeating this. 

[0003]In such a casesince the pattern with the same main performance patterns is 
repeatedthe performance tends to become monotonous. Thenthe sub patterns 
called a filHn or a breakad libetc. are preparedthe performance which follows 
these sub patterns according to directions by an artificial operation switch etc. is 
inserted temporarilyand it is performed that it is made to shift to the main 
patterns after that. Heresaid main patterns and each sub patterns are put in a 
database. 

It is made as [ reproduce / it is read by a player's operation and ]. 

[0004] Drawing 14 is a figure showing an example of transition of the performance 
pattern (style) in such an automatic performance. In the example shown in this 
flguretwo main patterns (a normal pattern and a variation pattern)the main A and 
the main BThe 1st and 2nd fill inn patterns respectively corresponding to each 
main patternThe main A as a fill inn pattern inserted during a performance 
Namelyan A->A filHn (FILL AA)And filHn A->B (FILL AB) when shifting to the 
main B from the main AThe main B as a fill inn pattern inserted during a 
performance A B->B filHn (FILL BB)And the B->A fill-in (FILL BA) which shifts 
to the main A from the main Bit has each performance pattern of the introduction 
pattern (INTRO AINTRO B) corresponding to each main patternand ending 
patterns (ENDING AENDING B). "FILL A" and two fill inn pattern selecting 
switches of "FILL B" which are operated when switching said each pattern 
(style)And each operation switch of "ENDING A" which chooses ending patterns 
and "ENID1NG B"INTRO Awhich chooses an introduction patternand "INTRO B" 



is formed. 

[0005]For examplewhen "INTRO A" is operatedafter the intro A is performed and 
the performance is completed firstthe performance of the main A is started. And 
when said FILL AA pattern is inserted and it returns to the main A after thatwhen 
the aforementioned "FILL A" switch is pushedand the aforementioned "FILL B" 
switch is pushed during the performance of the main Asaid FILL AB is inserted 
and it shifts to the main B after that. When "ENDING A" is pushedthe ending A is 
performed and the performance is ended. On the other handif "INTRO B" is 
operatedafter the intro B is performedthe performance of the main B will be 
started. And when the aforementioned "FILL A" switch is pushed during a main B 
performance. When said FILL BA pattern is insertedit shifts to the main A after 
that and the aforementioned "FILL B" switch is pushedsaid FILL BB pattern is 
inserted and it is made as [ return / to the main B / after that ]. If "ENDING B" is 
pushedthe ending B will be performed and the performance will be ended. 
[0006]Thusit is made as [ insert / according to a performance state when an 
operation switch is operated / the fill inn pattern corresponding to the main 
pattern under performance and the main pattern of the shifting destination / 
choose and ]and a performance can be prevented from becoming monotonous. 
Although the main pattern showed only two casesA and Bby above-mentioned 
drawing 14 it is not restricted to this. Various variations except having described 
above and the methods of transition are also known. For examplea fill-in can be 
performed only about some musical instruments of one part. 

[0007]About each note of not only the automatic accompaniment mentioned above 
but desired musicthe pitch of the notePronunciation start / silence start timing 
etc. are memorized as performance information in the form of SMF (Standard MIDI 
File) etc.for exampleand the automatic playing device which begins to read this 
performance information (music data) one by oneand generates musical tone is 
also known. A player may just merely operate a performance start switch and a 
performance end switch only. 
[0008] 

[Problem(s) to be Solved by the Invention]In the electrohone which can perform 
the automatic accompaniment and the automatic performance of the former which 
was mentioned abovewhen a player switched a performance patternthe interaction 
(interaction) by a sound could be performedbut the interaction by vision was not 
able to be performed. The indicator was provided in such electrohone and 
displaying the track name in an automatic performance or automatic 
accompaniment or displaying the vibrant tune and tempo at the time of a 
performance were performed. Displaying the key pushing directions which a player 
should do key pushing next by an indicator was also performed. Howeverdisplaying 
the performance itself on an indicator was not performed and seeing the situation 
of a performance was not realized. 

[0009]Thenan object of this invention is to provide the musical tone and the image 
generating device which can be performed seeing and enjoying various musical 
instrument playing by synchronizing with a performance the operation doubled with 



the performance styleand displaying it. 
[0010] 

[Means for Solving the Problem]To achieve the above objectsmusical tone and an 
image generating device of this inventionlt has an image generation part which 
generates a musical sound generation part which generates musical tone based on 
performance informationand image data which shows a situation of a performance 
corresponding to said performance information in a selected musical instrument or 
a part synchronizing with said performance information. Synchronizing with 
composition datait becomes possible to display a situation of the performance in 
arbitrary musical instruments or parts on an image display device by thisand the 
player can enjoy now not only an interaction by a sound but an interaction by 
vision. 

[001 1]It has the part database built with operation parts including performance 
information which memorized a locus of a playing action of a performance pattern 
in which it was subdivided for every musical instrument or every partsaid image 
generation part generates 3D animation image data corresponding to the 
performance information concerned based on information acquired by reading 
operation parts corresponding to said performance information from this part 
databaseand connecting these operation parts that carried out reading appearance 
one by one. Since operation parts are put in a databasewhile being able to use 
common operation parts to two or more patterns and musical piecesit becomes 
possible to add parts required at any time to a database. Thereforeit becomes 
possible to indicate the 3D animation picture by generation efficiently. 
[0012]Said operation parts have a pronunciation point marker in which timing of 
performance information showing a locus of a playing action of said subdivided 
performance pattern and pronunciation is shown. This becomes possible to use 
common operation parts to change of tempoetc.and size of a database can be 
made small. A highly precise synchronization with musical tone and a picture is 
attained by using this pronunciation point marker for a synchronization with a 
musical sound generation part. 

[0013]Change of a performance character and a viewpoint in said 3D animation 
image data generated is enabled by operator further again. Therebywhile the 
operator can enjoy variegated 3D animation picture nowit becomes possible to 
zoom in and to display a model performance of him. According to change of said 
performance character and a viewpointit has further again a means to amend said 
performance information. Therebysaid performance information can be made into 
information common to a performance character or a viewpointand it becomes 
possible to make size of a database small. 

[0014]Even if said image generation part is a time of tempo of musical tone 
generated based on said performance information being changedit is made further 
again as [ generate / said image data set constant / image restoration speed of a 
portion corresponding to pronunciation operation in the image data concerned ]. 
Therebyeven if it is a time of tempo being changeda picture in connection with 
pronunciation operation of a musical instrument is generated at the rate of 



usualand it becomes possible to reproduce a natural picture. Said image generation 
part is made further again as [ embellish / the image data to generate ] based on 
said performance information or a pronunciation state of a musical sound 
generation part. This becomes possible to express a shake state of a musical 
instrument corresponding to volume of musical toneenvelope information on a 
sound sourceetc.and a natural playing action can be expressed. Said image 
generation part is made as [ set / the number of reproduction frames ] for every 
said selected musical instrument or part further again. This becomes possible to 
change the number of reproduction frames with a solo player and a back player 
according to a change of a viewpointetc.for examplecorresponding to a musical 
piece to performand it becomes possible to reduce load of CPU. 
[0015] 

[Embodiment of the Invention] Drawing 1 is a block diagram showing the example of 
composition of the musical tone of this inventionand the 1 embodiment of an 
image generating device. A central processing unit (CPU) with which 1 controls 
operation of this whole device in this figureProgram storage which memorizes the 
control program with which 2 controls this musical tone and image generating 
deviceThe style database with which various automatic performance patternssuch 
as a rhythm pattern and an automatic base code patternwere stored 3The 
operation-parts database and scene part database for generating 3D picture for 
displaying the situation of a performanceWhile memorizing other various datathe 
memory storage which consists of ROMRAMetc. which are used as workspaceand 
4 are operation switch groups which consist of various kinds of handlers provided 
in the keyboard (keyboard) and the navigational panel. 5 is a sound source part and 
generates the musical sound signal of the scale sound for two or more 
channelsand a rhythm sound. This sound source part may be a thing of what kind 
of methodssuch as a waveform memory methodFM methoda physical model 
methoda harmonics composite systema formant composite systemand an analog 
synthesizer method of VCO+VCF+VCA. It is not restricted to the sound source 
circuit constituted using hardware for exclusive useand may be constituted by the 
program of the sound source circuit constituted using DSP and the micro 
programor CPU and software. The effect processing part for giving various kinds 
of effectssuch as a vibrato and RIBABUto the generated musical tone is also 
contained in this sound source part 5. 6 is a sound system for carrying out sound 
emission of the musical tone outputted from said sound source part 5. 
[001 6]7 is an image display device (graphic display)and it displays the situation of 
a performance of the selected musical instrument or a part with the animation 
picture of 3D while it displays the operating state of this musical tone and an 
image generating deviceand the operating condition of an operation switch. It is a 
MIDI interface circuit for 8 to perform external storagessuch as a hard diska 
floppy diskCD-ROMMOand DVDand for 9 perform communication with external 
MIDI apparatus. A video interface circuit for 10 to display the picture which shows 
the situation of said performance to the monitor 11 connected outsideand 12 are 
the buses for performing the data communications between said each component. 



[0017] Drawing 2 is a figure showing an example of the appearance of the musical 
tone of this inventionand the 1 embodiment of an image generating device. In this 
exampleas said operation switch group 4The start of the keyboard 40 and an 
automatic performance. The end of the start switch 41 to direct and an automatic 
performance. The situation of the musical instrument change switch 44 which 
chooses the style select switch 43 which chooses performance patternssuch as 
the stop switch 42 to directa rhythm which carries out an automatic 
performancethe mainand a variationthe musical instrument which displays the 
situation of a performanceor a partand a performance. When displaying the 
situation of the player change switch 45 which chooses whether the performance 
by what kind of character is displayed when displayingthe fill-in switch 46 which 
chooses the musical instrument which performs a fill— inand a performance. The 
viewpoint change switch 48 for determining a viewpoint when displaying the 
situation of the stage change switch 47 which chooses ******and a performance 
is formed. D in said musical instrument change switch 44the player change switch 
45and the fill— in switch 46GBand K herelt is a switch for choosing a drum parta 
guitar parta base partand a keyboard partrespectivelyand A-D is a switch which 
chooses the details in a part with said DGBand selected Krespectively. In this 
examplethe situation of the performance of two or more parts (a keyboarda 
baseand three parts of a drum) is displayed on the image display device 7 (or 
external monitor 11) with 3D animation picture. 

[0018]Thusbefore explaining the processing which displays 3D animation 
picturesaid operation-parts database 20 is explained first. This operation-parts 
database 20 for every various musical instruments or every part. Subdivide various 
performance patternsrespectivelyand while incorporatingfor example as motion 
capture data and decomposing into xyand z shaft orientationsthe playing action of 
the this subdivided patternit is this ******** about pronouncing timing (for 
examplethe case of a drum RBI position) — marking is carried out to data and it 
puts in a database. The data of the these-subdivided playing action is called 
operation parts. As drawing 3 is a figure showing an example of the operation parts 
of a drum part and it is shown in this figureEach operation parts are constituted by 
the data of the short phrase ABand C and D by which the drum part was 
subdividedthe performance information which shows the locus of the operation of 
the player at the time of the pattern performance to every performance pattern of 
— and said pronunciation point markerand it is stored in said operation-parts 
database 20. In this examplealthough performance information of three musical 
instrumentscymbala snare drumand a bass drumis used as one operation partsin 
the case of musical instrumentssuch as a piano and saxoperation parts are 
generated for every musical instrument. 

[0019]The processing which creates said operation parts is explained with 
reference to the flow chart of drawing 4 . Firstin Step S10the state where the 
player is performing the specific phrase subdivided by the specific musical 
instrument is acquired as motion capture data, (a) of drawing 5 is a figure for 
explaining that situationas shown in this figureit equips a musical instrument with 



3D digitizer [ a bodily important section and if needed ] for a playerhas said 
subdivided specific phrase performedand records a motion of the body of the 
player at that time. The magnetic thing or the optical thing is known as a 3D 
digitizer. And in Step S1 1 in the motion capture data which was carried out in this 
way and acquiredthe locus of the central point of each part is decomposed into a 
xyz coordinate systemand the performance information which shows the 
movement state and position of each part is acquired. Temporal data may also be 
simultaneously recorded at this time. 

[0020]Nextit progresses to Step S12 and it is remembered as a marker (it is called 
a pronunciation point marker) that the coordinates of the key-point part of the 
position (pronunciation point) from which pronunciation took placeand the lapsed 
time from the performance start can be distinguished. For examplewhen it is the 
performance of the phrase shown in (b) of drawing St hree positions shown in a 
figure serve as a pronunciation pointand each lapsed time tt'and t" are memorized 
so that distinction is possible. As long as this pronunciation point marker can 
specify now the data corresponding to a pronunciation point within a set of said 
acquired performance information datait may be a thing of what kind of form. And 
it progresses to Step S13 and matching with the data acquired as mentioned 
above and the performed phrase is performed. And it puts in a database as data of 
form which can respond to change (change of the shape of a player and a musical 
instrumentor a size) of the position at the time of reproductionor change (change 
of tempo) of time. It may be made to also include each data of xyz coordinates 
and time and the data of the movement speed for every partaccelerationetc. 
besides a pronunciation point marker which were mentioned above as said 
operation-parts data. 

[0021 ]The processing which indicates the 3D animation picture by generation 
using the operation-parts database 20 created by carrying out such is explained 
taking the case of the case of the device which has an automatic accompaniment 
function. Drawing 6 is a figure showing the flow of the processing in this automatic 
accompaniment reproductionand shows the flow of regeneration of 3D animation 
picture in which the musical tone of one part and the situation of a performance 
are shown. What is necessary is to perform processing shown in this drawing 6 
about each partto compound themand just to displaywhen the situation of a 
performance of two or more parts is shown. 

[0022] Firstif the operation switch group 4 mentioned above is operated by the 
player and automatic accompaniment control operation is performedprocessing of 
Step S20 will be performed. In this step S20the pattern which should be 
reproduced is chosen from said style database 21 according to the operation made 
by the player. This is the same processing as a case with the conventional 
automatic accompaniment function mentioned above. Thusselected performance 
style data is handed over by each processing of Step S21 and Step S25. 
[0023]Step S25 is the conventional automatic accompaniment processing 
mentioned above and the same processingand generates pronunciation eventssuch 
as a key ON event and a control changeand a sound-source-control parameter 



based on the performance information included in selected performance style data. 
Thusit is inputted into the sound source part 5corresponding musical tone is 
generated (Step S26)and the generated sound-source-control parameter is 
outputted from said sound system 6. 

[0024]On the other handin Step S21 corresponding operation parts are chosen 
from said operation-parts database 20 based on said selected performance style 
dataand basic motion information is generated. Herein the case of automatic 
accompanimentsince the operation parts corresponding to each performance style 
can be known beforehandthe information which specifies operation parts [ be / it / 
under / said selected performance style data / correspondence ] can be included. 
[0025]An example of generation of this basic motion information is explained using 
drawing 7 . (a) of drawing 7 is an example of the phrase corresponding to each 
operation parts stored in said operation-parts database 20. That isthe operation 
parts respectively corresponding to the phrase ABand C shown in this figureDand 
— are put in a database and stored in said operation-parts database 20. And the 
performance pattern corresponding to the selected performance style 
presupposes that it is the pattern shown in (b) of drawing 7 . In this casein this 
step S21the operation parts according to said pattern are read from said 
operation-parts database 20. And basic motion information is generated by piling 
up the portion of the end of each operation partsand the head part of the 
operation parts which followand connecting and uniting them. As a resultabout the 
basic pattern shown in (b) of drawing 7 operation parts corresponding like A->B- 
>C->B will be connected. 

[0026]When variation operation of the fill-in to the specific musical instrument 
mentioned aboveetc. is madeit progresses to Step S22 and processing which piles 
up or substitutes the performance information according to a fill-in is performed 
to the basic motion information generated in said step S21. For examplea variation 
pattern as the style pattern which should be performed shows to (c) of drawing 
6Namelyin being what performs a fill-in about the cymbal of the drum partsand a 
snare drum. By changing the portion of the last of the basic motion information 
(A->B->C->B) generated in said step S21and the data in front of one of them to 
the data of the operation parts Dthe performance information corresponding to 
this variation pattern can be acquired. Thusby substituting the part of the 
operation parts for some other partsit can respond to the variation operation 
mentioned above. 

[0027]Nextthe display part select data which progresses to Step S23 and is set up 
by said alter operation child 44The player character select data from said alter 
operation child 45the viewpoint change manipulation data from said alter operation 
child 48While choosing and reading the information corresponding to these from 
the scene part database 22 according to the stage change manipulation data from 
said alter operation child 47etc.the compensation process of the coordinate data 
contained in said performance information based on these information is 
performed. That isthe scene parts corresponding to the part which displays a 
performance state or a musical instrumentthe character which is carrying out the 



performancethe selected stageand the specified viewpoint (camera position) are 
read from said scene part database 22. When the display of two or more parts or 
the performance state of a musical instrument is directedthe scene parts 
corresponding to those arrangement are read. 

[0028]With reference to drawing 8 an example of the compensation process of said 
coordinate data is explained. This example shows the case where the musical 
instrument which should display the state of a performance is cymbaland assumes 
that the locus of the stick shown by (1) from an initial position (xOyOzO) to the 
target positions (xtytzt) on cymbal is contained in performance information. At this 
timethe height of cymbal is changed with the data of a player character or view 
position data with the selected operatorand suppose that the coordinates of the 
target became (xt'yt'zt'). At this timein this step S23said performance information 
is amended so that it may become a locus shown by (2). When a player is changed 
and the initial position of said stick is changed into the position shown with the 
dashed line in a figurePerformance information is amended so that it may become 
a locus shown by (3)and furtherwhen a player and both of the height of cymbal are 
changedthe compensation process of said performance information is performed 
so that it may become a locus shown by (4). Thusin this step S23a model position 
is determined and animation is determined that it will correspond to it. 
[0029]To each operation parts stored in the operation-parts database 20 in this 
invention at this time as mentioned above. Not only the coordinate data in 
alignment with a time-axis but the pronunciation point marker is includedand the 
time or speed to a pronunciation point can be acquired from the coordinates of 
each pronunciation pointand the reproduction start of that performance 
information with this pronunciation point marker. Thereforehe is trying to take the 
synchronization with the image to generate and the musical tone to generate 
based on this pronunciation point marker. 

[0030]namely — said — drawing 5 — ( — b — ) — having been shown — as — 
each — pronunciation — a point — up to — reference tempo — it can set — 
time — t — t — ' — t — " — said — operation parts — from — being 
acquirable . Thereforewhen the tempo performed to tempo (reference tempo) 
when said operation parts are created is changed into k times as much tempo, 
operating performance information read-out on a curtailed schedule so that it may 
reach from the reproduction start of said performance information to a 
pronunciation point in 1/k time as much time (one the speed [ If it is speed ] of k 
times of this)and the reproduction interval of the performance information may 
become short or long **** — multiple times — what is necessary is just to 
control like reading the same active position When transit time or movement speed 
is prepared for every coordinates (i.e.when time until it moves to the following 
coordinates from coordinates with each partor the information on speed is 
included in operation parts)What is necessary is to change k timesrespectively at 
the time of 1/k time and speed (amendment)and just to controlwhen it is time. 
[0031 ]By the wayit may become an unnatural picture only by controlling a time- 
axis simply as mentioned above about all the operations. For exampleif tempo is 



made into a halfon the wholethe working speed of a picture will turn into a half 
speedand in a performance of a drum etc.it will be a picture which is struck quietly 
and will be visible like the performance which stopped volume. After starting 
operationit enables it to recognize the part about pronunciation operation in 
operation to a pronunciation point (from a pronunciation operation start point to 
the silence operating point)and even if it changes tempoit is made not to change 
the working speed from a pronunciation operation start point to the silence 
operating pointin order to avoid this. 

[0032]This situation is explained using drawing 9 . In this flgurea horizontal axis 
shows time and the vertical axis shows the playing action (for exampleposition of a 
drum stick). As shown in (a) of a figurein the case of the usual tempomovement of 
a drum stick was started from the pronunciation operation start pointthe lowest 
position was reached in the pronouncing pointand it has returned to the position of 
the basis in the silence operating point again, (b) shows the case where a time- 
axis is elongated simplywhen tempo is changed lateand it is longer [ the time by 
the pronouncing / pronunciation operation start point -> / point -> silence 
operating point ] than the case of (a). Thena pronunciation operation start point 
reproduces a picture at the speed corresponding to tempoand it is made to be 
reproduced from a pronunciation operation start point at the same speed as the 
case of the above (a) till the silence operating pointas shown in (c). Therebythe 
same picture as the actual performance corresponding to volume can be displayed. 
Also when tempo is carried out earlyit is made not to change the reproduction 
speed from a pronunciation operation start point to the silence operating point 
similarly. 

[0033]It is made to embellish performance information using sound-source-control 
parameters generated in said sound-source-control parameter generation step 
S25 ( drawing 6 )such as an envelope and a velocity. Shake operation of cymbal is 
taken for an example and this is explained with reference to drawing 10 . (a) of 
drawing 10 is a figure showing the appearance of a shake of cymbal. Shake 
operation of the standard cymbal created by motion capture is stored in said 
operation-parts database 20 as mentioned above. Although the performance 
information which expresses the shake of cymbal synchronizing with the 
pronouncing timing in musical tone is generated based on these operation partsAt 
this timebased on musical-sound-control parameterssuch as a velocity or track 
volumewhen a velocity or track volume is largethe shake of cymbal is enlargedand 
performance information is generated so that a shake smallif conversely small. 
[0034]That isthe maximum volume is set to M and volume at the time of 
pronunciation is set to m. The cycle of a shake assumes that it is expressed by 
the sine wave. When the maximum amplitude value of a shake of cymbal is set to 
A herethe shake of the cymbal after pronunciation is expressed with the following 
formula (1). 

The amount of shakes = (m/M) -A-sint — (1) 

Heret expresses time. The maximum envelope value is set to Ethe envelope value 
e of the channel which has pronounced cymbal by the sound-source channel after 



that is readand this is used as input. In this casethe following formula (2) can 

express the amount of shakes of cymbal. 

The amount of shakes = (m/M)(e/E)and A-sint — (2) 

That isaccording to attenuation of the envelope value e of a sound sourcewhile 
the amount of shakes of cymbal also repeats a periodic motionit decreases, (b) of 
drawing 10 is a figure showing this situation. Thusthe quality picture corresponding 
to volume can be created by using the computed amount of shakes for the 
correcting operation of performance information. 

[0035]The reproduction frame rate of a picture can also be changed per a player 
unit or part. For examplea reproduction frame rate can be dropped about a back 
player or the player which moved back by the viewpoint change. Drawing 1 1 is a 
figure showing this situationand is explained taking the case of the case where the 
picture of a solo part's player and two players (the back 1 and back 2) of a back 
part is reproduced as shown in (a). In this caseas shown in (b)a solo player 
generates a picture by the usual reproduction frame rate (for example30 frames 
per second)and generates a picture by turnsfor example by the frame rate of 
1/that 2 about the player of the back 1 and the back 2. Therebyit becomes 
possible to decrease the arithmetic load of CPU. 
[0036]If it reports having reached image generation processing of the 
pronunciation point to said sound-source-control parameter generation step S25 
from the image processing portion S24 further again and is made to perform 
pronunciation processing in said S26 based on thislt becomes possible to raise the 
synchronous certainty of pronunciation and the picture generated. Thusit becomes 
possible to generate the performance picture which has a right pronunciation point 
according to the tempo of a performance. 

[0037]Nextit progresses to Step S24 and image generation processing (rendering) 
is performed using the information determined by said step S23. That isconversion 
to signals of a scene is performed based on said scene information and 
performance information, namely — being based on said scene information or 
performance information — coordinate conversionhidden surface removalan 
intersectionand a nodal line — calculation of - intersection side etc.shadingtexture 
mappingetc. being processedand the luminance value of each pixel of the picture 
on an image display device[ output and ] 3D animation picture is generated and it 
outputs to the image display device 7. Synchronizing with automatic 
accompaniment datathe state of a performance of arbitrary parts can be displayed 
with 3D animation picture as mentioned above. 

[0038]Nextthe embodiment of the invention applied to the automatic playing 
device which reproduces the music data of desired music is described with 
reference to the flow chart of drawing 12 . When performing such an automatic 
performancethe performance information (music data) of the musical piece which 
should be performed is stored in the music database 23. In Step S30selection of 
the music which carries out [ operator ] an automatic performance will read the 
regenerative data of the music chosen from said this music database 23 specified 
length every. And this regenerative data is given to Step S31 and Step S34. Step 



S34 and S35 are the same processings as Step S25 in the case of said automatic 
accompanimentand S26generate the musical tone based on regenerative dataand 
output it from the sound system 6. 

[0039]Steps S31-S33 are processings which generate 3D animation picture 
corresponding to regenerative data. In Step S3 1 the operation parts nearest to the 
regenerative data of said specified length are chosen and read from said 
operation-parts database 20. And basic motion information is generated by piling 
upconnecting and uniting the portion of the end of each read operation partsand 
the head part of the operation parts which follow as well as said step S21. That 
isthe length (it is called the 1st portion) corresponding to the phrase subdivided 
from the beginning of playing data is taken outand the operation parts 
corresponding to this and the nearest phrase are read from said operation-parts 
database 20. Nextthe 2nd portion is similarly taken out for the end of said 1st 
taken-out portion as a headthe operation parts nearest to this are read from said 
operation-parts database 20and it connects with the operation parts taken out to 
said beginning. Basic motion information is generated by connecting like the 
following the operation parts which repeat the above-mentioned operation and 
correspond. Although an example which the above diverts the operation parts 
prepared general-purposeand is adapted to arbitrary music data is 
shownOperation parts are formed into a standard-basis book set (for examplethe 
fundamental tone color is automatically matched with the timbre number like the 
GM fundamental tone color like)It may be made to put the operation-parts 
directions information corresponding to the applicable operation parts of said basic 
set which should be used within music data according to advance of music in 
music data. 

[0040]And it progresses to Step S32 and the determination of a model position 
and a decision of animation are made like said step S23. And it progresses to Step 
S333D animation picture corresponding to performance information is generated 
like said step S24and it is made to display on the display 7. Thusalso in the case 
of an automatic performance3D animation picture showing the performance state 
of the musical piece can be displayed. 

[0041]Nextother examples of the musical tone of this invention and the 
appearance of an image generating device are shown in drawing 13 . Signs that 
each handler is arranged at the right and left of the image display device 7and the 
example shown in this figure is performing one part (this example drum part) on a 
screen are expressed as 3D animation. The start button of an automatic 
performance and 52 here the handler 51 The stop button of an automatic 
performanceA tempo rise button for 53 to make the tempo of a performance 
raising and 54 A tempo down buttonlt is a musical instrument selection button for 
carrying out selection decision of of which musical instrument the player selection 
button for choosing a player in case 55 displays a performance state on the image 
display device 7and 56 display a performance statewhen displaying a performance 
state similarly. 57 and 58 are the buttons for choosing the main pattern (main 
style) of an automatic performanceand the main A button in which 57 chooses the 



main Aand 58 are main B buttons which choose the main B. A fill inn button for an 
intro button for 59 to choose the pattern of an intro and 60 to insert a fill inn 
pattern and 61 are the ending buttons for choosing ending patterns. 62 is a view 
point movement button for moving the viewpoint in the case of displaying the 
performance state of 3D to the image display device 7 mentioned above further 
again. 

[0042]As mentioned abovethe picture displayed in this invention can be displayed 
even if it is any of one part or two or more parts. It is clear that it is applicable to 
a sequencer without a keyboard partetc. In the above explanationalthough 
explained taking the case of the case of automatic accompaniment or an 
automatic performance3D animation picture can be similarly displayed to the 
playing data of a melody part inputted by placing etc. 

[0043]According to the stage selected with said stage change switch 47the effect 
given in said sound source part 5 can be changed. For examplewhen a hole is 
chosendelay is enlargedand an effect can be made to change according to the 
situation of said picture displayed as delay is lessenedwhen the outdoors is 
chosen. Although explained taking the case of the case where performance 
information (motion file) is acquired with motion captureit may be made to 
generate performance information by methods other than motion capture like a 
key frame method in the above. 
[0044] 

[Effect of the Invention]As explained aboveaccording to this invention3D animation 
picture can be displayed synchronizing with composition data. Thereforethe 
operator can enjoy not only the interaction by the sound according to a codea fill- 
inetc. but the interaction by 3D animation picture. Since operation parts are put in 
a databasewhile being able to use operation parts in common to two or more 
patterns and musical piecesrequired parts can be added to a database. Thereforeit 
becomes possible to generate 3D animation picture efficiently. Since it has the 
pronunciation point marker with performance information as operation partsit 
becomes possible to use common operation parts to change of tempoetc.and it 
becomes possible to make size of a database small. The picture which does not 
have further again the unnaturalness which synchronized with the performance of 
musical tone can be displayed. The operator can choose the character suitable for 
liking of him from two or more characters further again. Since the operator can 
change the position of the viewpoint of a display imageit becomes possible to see 
the model performance state seen from arbitrary positionsand he can use it also 
for an educational use further again. 
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[Brief Description of the Drawings] 

[Drawing 1] It is a block diagram showing the example of composition of the 
musical tone of this inventionand the 1 embodiment of an image generating device. 



[Drawing 2] It is an outline view of the musical tone of this inventionand the 1 
embodiment of an image generating device. 

[Drawing 3] It is a figure for explaining an operation-parts database. 

[Drawing 4] It is a flow chart for explaining generation of operation parts. 

[Drawing 5] It is a figure for explaining generation of operation parts. 

[Drawing 6] It is a flow chart of image generation display processing and musical 

tone generation processing in automatic accompaniment. 

[Drawing 7] It is a figure for explaining creation of basic motion information. 

[Drawing 8] It is a figure for explaining coordinate correction processing. 

[Drawing 9] It is a figure for explaining the processing at the time of changing 

tempo. 

[Drawing 10] It is a figure for explaining ornamentation processing of operation of a 
musical instrument. 

[Drawing 1 1] It is a figure for explaining the change processing of a reproduction 
frame rate. 

[Drawing 12] It is a flow chart of image generation display processing and musical 

tone generation processing in an automatic performance. 

[Drawing 13] It is a figure showing other examples of the musical tone of this 

inventionand the appearance of an image generating device. 

[Drawing 14] It is a figure showing the example of transition of the performance 

pattern in automatic accompaniment processing. 

[Description of Notations] 

1 CPU2 program storageand 3 Memory storage4 operation switch groupsFive 
sound source parts and 6 A sound system and 7 An image display device and 8 
External storageNine MIDI interface circuits and 10 Video interface circuitl 1 A 
monitor and 20 An operation-parts database and 21 Style database22 scene part 
database23 music databasesand 40 Keyboard41 A start switch and 42 [ A player 
change switch and 46 / A fill-in switch and 47 / A stage change switch and 48 / 
Viewpoint change switch ] A stop switch and 43 A style select switch and 44 A 
musical instrument change switch and 45 



